The transaction would give the lingerie brand a valuation of about $1.1 billion, according to the Wall Street Journal.
Sycamore Partners is expected to take the brand private, buying 55 percent.
L Brands would keep a 45 percent stake in the separate company.
The Bath & Body Works chain would be the only remaining name under the L Brands umbrella.
Victoria's Secret, known for its supermodel "Angels" wearing padded bras, has struggled in recent years with falling sales.
Last year, Victoria's Secret canceled its televised fashion show.
The deal would call for Leslie Wexner, the billionaire who has run the company for more than 50 years, to step down from his roles as CEO and chairman, according to the Journal.
He'll remain on the L Brands board and hold stakes in both companies.
He currently owns about 17 percent of L Brands shares.
L Brands ended Wednesday with a market capitalization of around $7 billion, down from $29 billion at its peak in 2015.
Sycamore Partners would add the lingerie brand to a roster of companies it has purchased including The Limited, Hot Topic, Nine West and Staples.