The rapper’s second solo album Set It Off was released on Oct. 13

A fan-favorite song on Offset’s new album was almost left off the tracklist.

Offset released his second solo studio album Set It Off earlier this month, and it features a variety of artists throughout the 21 tracks. However, the former Migos member tells PEOPLE that “Worth It” with Don Toliver was not only his favorite to record, but it also wasn’t on the original final version of the album.

“I got that song two weeks before the album came [out],” Offset, 31, said while chatting about his new collab with Howlin’ Rays and Postmates. “The album was already done and I felt like I had everything, and then he came to the studio in L.A. and played me the record and I’m like, ‘Yo, what is this?’”

He continued: “I Immediately got attached to it and I cut my verse and it was like a fourth quarter — it was like a fourth quarter buzzer beater.”

Offset; Don Toliver. KEVIN MAZUR/GETTY; MONICA SCHIPPER/GETTY

In the making of Set It Off — which took two-and-a-half years —Offset says he was ultra-critical of himself and his first album in four years. His goal, he explains, was to share voices with his fanbase that he hadn’t in the past.

“Of course people are familiar with me working with Travis [Scott] and Future, but I had not done but one song with Don Toliver — ‘Had Enough’ — and it went crazy on his first album,” he says. “I just wanted to let other people hear me with other people instead of it being the same track listing — basically the same features I went through my whole career.”

Offset; Don Toliver. PIERRE SUU/GETTY; JOHN NACION/WIREIMAGE

Out of all the artists he worked with, the “Clout” rapper says he wanted to “shout out all the women on the album.” Latto, Chloe Bailey and his wife Cardi B all had features, and Offset reveals he was impressed with their dedication.

“Sometimes you do features and they just kind of do it real quick, but I could tell from the artistry of those three women that it was like, they took that s— serious,” he raves.

Offset; Chloe Bailey. TAYLOR HILL/WIREIMAGE; JAMIE MCCARTHY/WIREIMAGE

Now that he is making music on his own, Offset says he hopes to go on tour as soon as this spring. His “dream,” he says, is to “become a big tour artist” eventually.

“I’m going to start with 2000 seaters because I don’t want to get caught up in the ego,” he tells PEOPLE. “I just want to build my solid fan base.”

Offset’s Set It Off. OFFSET INSTAGRAM

Calling it his “identification album,” Offset says he is happy with the response he has gotten since the release of Set It Off, and sees it as the start of his solo career.

“People enjoy and also get the storyline of what I’m saying in my music,” he says, adding: “I wanted to make sure that I came with something new that didn’t sound like any of my old stuff.”

Angelenos can order the Howlin Ween Offset Shake Box ($15) exclusively from Postmates beginning Oct. 25 while supplies last.