WantToKnow.info's New Archive Search Page

in #life4 days ago

photo_2024-12-11_18-49-44.jpg

The image above was made by @amberjyang with Midjourney using the prompt 'shoveling or pitchforking a pile of computer code.'

WantToKnow.info probably has the best archive of conspiracy news on the internet. Our database contains over 13k news article summaries. Our main site makes it easy to search this database and a specialized search app I made last spring makes it possible to explore the data even more thoroughly. Now I've created a public facing database interface to make it easier for researchers to look through vast numbers of entries very quickly.

I used one of our admin pages as the basis for the new page. This code was largely foreign to me when I got started. It was spread across dozens of files and was applying css from 4 sources. The main files were php, which I'm only vaguely familiar with. Building the new page on this foundation was a fun challenge.

After I got the interface working, I sent it to my colleagues, who immediately found several show-stopping bugs. Fortunately, I was able to figure all of these out and correct them fairly quickly. If you go to the page and find a bug, please comment it below. If there's something you think should be improved, comment that too.

For years, I resisted the idea of working on our website directly, preferring instead to just edit the news. But the web developer who did most of the work on the site recently moved on to another job. Since our dev needs are minimal, I volunteered to step into the role for now. This is my first project in the new role and I feel like it went okay.

The hardest part was getting over my hangups about how things should be. In my opinion, our database should just be a json file or there should be an api layer between the sql database and our applications. Our code should be well-commented and maybe even documented somewhere. Variable names should be unambiguous and unique enough to avoid collisions. Changing a table's column width should take under 5 minutes and not cause the display of search results to crash.

Once I let go of these hangups, the work was an enjoyable puzzle. While it took longer than I would've liked, it also nudged me out of my comfort zone in a good way. My level of competence with php remains low, but now I know better what I'm looking at when I see it. I also have a better idea of what it would take to integrate vector search functionality into this interface.

It might someday be cool to combine our various search tools into one super slick search + reading app. Ideally, this would be able to search all of our pages, not just our news articles archive. To get there might require writing a crawler and calculating vectors, then storing these somewhere. This is totally doable, but it'll take some time.

I think of myself more as a data science hobbyist than as a developer. Coming from a python background, javascript still just looks kind of messy to me. But this matters less and less in the dawning age of AI. And every little project I take on gives me more reason to interact with the latest iteration of the tech.

Within a few years, I expect AI to be able to simply spit out any kind of custom website a person might want. It can already format and comment code passably, though trusting it with anything more complex is risky. I look forward to the day when AI can take all of the busywork out of coding, so I can focus completely on the creativity of it. The day when I can describe an algorithm in a paragraph and be presented with a perfectly crafted component in the language of my choosing. This is my nerd fantasy.


Read Free Mind Gazette on Substack

Read my novels:

See my NFTs:

  • Small Gods of Time Travel is a 41 piece Tezos NFT collection on Objkt that goes with my book by the same name.
  • History and the Machine is a 20 piece Tezos NFT collection on Objkt based on my series of oil paintings of interesting people from history.
  • Artifacts of Mind Control is a 15 piece Tezos NFT collection on Objkt based on declassified CIA documents from the MKULTRA program.