(CNN) -- Attorney General Eric Holder is spending his remaining time in office as an activist for civil rights and social justice issues, ranging from gay rights to changing sentencing laws for non-violent crimes.

If all goes as planned, it could help change his legacy.

The key mile-marker in Holder's five-year tenure is President Barack Obama's February 2011 decision -- with Holder's recommendation -- to quit legally defending the Clinton-era Defense of Marriage Act. The law required the federal government to deny recognition of same-sex marriages.

That decision, which came at a time when the President said he was still "evolving" on whether to support such marriages, eventually led to the Supreme Court's ruling last year that overturned the law.

And it led to Holder's announcement in recent weeks to extend spousal benefits to same-sex married couples in all federal legal matters.

For Holder, 63, it's a way to carve out a modern civil rights legacy.

In discussing his gay-rights efforts, he often mentions his sister-in-law, who in 1963 made history as one of the two black students who integrated the University of Alabama under federal armed guard.

A portrait of a Justice Department prosecutor who stood watch on the university steps that day hangs in Holder's personal office in a fifth-floor corner of the Justice Department headquarters named for Robert F. Kennedy.

Holder and his supporters have eagerly embraced the RFK legacy as a comparison.

"When you look at his entire record, Eric Holder will have done more to expand justice in the United States since Bobby Kennedy," says Matthew Miller, a former aide to Holder who remains close to the attorney general.

It's not the way his critics have portrayed him, particularly during some of the rocky periods of his tenure.

For a time, he was involved in fights with Republicans and White House rivals over a variety of national security issues.

And he survived a bruising battle with House Republicans over the "Fast and Furious" gun trafficking controversy, including becoming the first sitting Cabinet member to be held in contempt of Congress.

"I've had some days that have been better than others," Holder told CNN in a November interview. But added that now, "There's the ability to focus on issues now that really matter to me."

Among those issues is working to change sentencing guidelines to give prosecutors more flexibility in certain non-violent criminal cases, pushing for rehabilitation instead of warehousing of prisoners, and getting rid of the disparity in the way the justice system deals with defendants charged in cocaine and crack cocaine crimes.

Holder also recently made a public push for states to more quickly restore voting rights to felons after they've completed their sentences and probation.

He is joined in that effort by Sen. Rand Paul and other conservatives who've similarly view it in line with their views on expanded individual rights.

Following Holder's plea, Alabama's Republican governor said he could support such an idea, and a modified proposal in Paul's home state of Kentucky has begun moving through the legislature.

But Holder's moves to use administrative memos to make effective changes before Congress passes new laws have also generated criticism.

Former Attorney General Michael Mukasey told CNN's Jake Tapper in an August interview that "I generally agree with the goal of getting rid of mandatory minimums. ... But the way to do that is to pass a law, not to say you're going to disregard the law."

Sen. Charles Grassley similarly has criticized Holder and the Obama administration for taking action without waiting for Congress.

But the department's decision to not defend the federal ban on same-sex marriage has since become emulated in states, including Virginia and Oregon. Those moves could propel the issue back to the Supreme Court in the next two years.

Holder's most aggressive moves have come in recent weeks with an aggressive interpretation of last year's Supreme Court ruling on same-sex marriage.

After much internal debate, Holder decided the department would give full equal treatment to same-sex married couples, going so far as extending the right to refuse to provide incriminating testimony against a spouse, a bedrock right for married couples.

Earlier this month, Holder basked in applause in a New York City hotel ballroom where he announced the department's latest decision on same-same marriage legal rights.

The Human Rights Campaign, which held the event, praised him, by saying the decision "cements his place in history alongside Robert F. Kennedy, another attorney general who crusaded for civil rights."