First, don't feel guilty. You did your brother a favor by lending him money at a time when he was in need -- and now it's time for him to be responsible about paying you back. Of course, he may not be thrilled to be reminded of his obligation, but you'll certainly feel some resentment yourself if you pretend the loan doesn't exist. Open, forthright communication is generally always better than avoidance.
Second, think about whether you can be flexible. Maybe it is taking him longer to regain his financial footing; he may need some extra time to start paying at a lower rate or a longer term. If that's the case, see what you can work out. If you agree to change the terms of the loan, document the changes in a new loan agreement.
But being flexible doesn't mean being a pushover. I think it's vital for both of you that he honor his obligation, even if on more lenient terms than originally outlined. That way, this transaction will cease to be a bone of contention between the two of you.