This
easyphp setup guide is for a couple versions back of easyphp, but it honestly hasn't changed, at least not much on the surface IMO.
There are countless php / mysql tutorials online, but I found that
tizag's tutorials are pretty good.
As far as a walk-through for just phpmyadmin, I don't know. It always seemed pretty self-explanatory. Maybe it's just you need to know the basics to get it. You can think of a mysql database like a single excel file. Inside a db you have tables (equivalent to sheets in excel) and in those tables you have fields (columns in excel) and records (rows in excel). Beyond that it can get a little complicated with optimal field types and rules, collation, etc, but if you just want to fool around with the ootp db in phpmyadmin you probably don't need to know all that advanced stuff.
The hardest part is probably just setting it up (easyphp makes this, well, easy) and importing the db (click on the import tab in phpmyadmin and it's pretty easy from there). After that you can just browse it.