@0x4d6165@wanderingwires.netwould anyone be willing to help me brainstorm about the feed reader? either on here or off platform. I'm trying to figure out a few things, but right now I'm focusing on respecting robots.txt. I need a way to check when the last time I accessed a host was so that I don't hit them with a request before the crawl-delay has elapsed. Would just a database table with the hostname and the last access timestamp be a good solution? I've considered using redis/valkey as well with persistence on.