There is no denying that the web has changed. I got onto the internet in the late 90's; relatively late in the game but ancient by today's standards. This was the early days of the world wide web, but late enough that things weren't completely bare bones. (I had used things like Gopher very early on in school and knew people who used services like Compuserve, but it was only in the late 90's that I had the internet at home.) When I compare those days to today, it feels like a completely different entity.
We can definitely blame smart phones for a lot of the fallout. At first we got "mobile friendly" sites, but this style quickly became the default. Infinite scrolling in a small column. Everything cramped. The smartphone user was much less of an "explorer" of the web, and so things got more and more centralized. Most people can "browse the web" literally all day, but never leave youtube and twitter. In contrast in the late 90's or early 00's if you were browsing the web you were going from website to website. There were things like fanfiction archives, and by the mid 00's some sites had a lot of video content (for exmaple the pre-youtube AVGN or ThatGuyWithTheGlasses sites; and of course we had stuff like Newgrounds and YTMND.) But you couldn't really spend the whole day at those sites, especially if you wanted to interact with people. Now it's very rare that you are linked to a video without it being on youtube, or maybe one of the few competitors. (It's not like I have any plans to host videos directly on site here; that's just the way things are now.)
What I want to talk about in this essay is what a different experience this meant for web browsing. In those days it felt like you could end up anywhere when you browsed the internet. There's a very different feeling in going to some random guy's personal site, set up just like how he wanted it, when compared to finding an obscure video on youtube. It's not just finding an unusual piece of content; it feels like you went to a different place. It's kind of like the difference between seeing a new product at Walmart vs. ending up in a locally run specialty store in some small town.
Back then you could easily run into these places just by opening a search engine, or using a web directoy (back when those were in still common use.) I noted in this essay that back in 1997, you could go to Yahoo!, start digging into the science fiction section, and end up on some guy's page about Star Trek. You could literally buy directory CDs in stores that would have dozens if not hundreds of personal fansites (as well as many Usenet channels.) Now if you search for anything you are unlikely to escape youtube, wikipedia, twitter and reddit. The curated search Wiby does a good job of finding interesting sites, but they are mainly old and abandoned. The site Curlie is a promising web directory, but at the moment it is still very barebones (and always seems like it is in the danger of dying.) Now I do love both of those sites, I just want you to be aware of their limitations. Back in the day you could get similar, but much more extensive, results using literally any search engine or directory.