Can we be friends?

My team at work which composes of a total of 4 are all pretty young meaning the oldest is 40.  I'm the youngest in the group and also single.  Everyone else is married.  For the most part I don't feel out of place when it comes to tv shows, current gossip, and things we do on the weekends.  Well for me I seem to be the entertaining one with stories to tell.  I can't even say stories it's more of me generally speaking and they delve deeper into what I'm discussing.  So the other day I kept referring to my happy hour friend.  Then my boss asks if he' s a guy.  I thought does it matter?  Which I responded just like that and then everyone assumed he's a guy which they were right.  I guess I was a bit reluctant about it because I already know where they're going with this.  How come you're just friends, how did you meet, etc.etc.  Well the kicker is I told them it's just a happy hour buddy who I meet on occasion who happens to be married.  Yes he is married and he's not hiding it.  He and I already had the talk of what his intentions are with me and why we hang out.  I felt like there are no lines crossed and we can be friends.  Until he crosses that line then our "friendship" would end.  Plain and simple.  They were all condemning me for hanging out with a married man and saying I'm setting up the foundation for something to happen, which I totally understand and not naive to know the potential damage....not in my life but his if his wife knew.  But then again what does she need to know that he's hanging out with a single gal.  There's plenty of single women at happy hour.  We happen to meet same time same place nothing more nothing less.  I pay for my meals he pays for his.  I see nothing wrong.  What I did realize after my boss pointed out is what am I getting out of it.  His benefit is that another women is giving him attention and the possibility of what may happen is exciting to him and this drives him to meet me every time. The excitement of getting closer and closer can lead to something.  I get that he's a guy so he hasn't put much thought into our friendship cuz I haven't either.  But I did notice that every time I do see him he's getting to know a little bit more of me, more to where I'm getting comfortable and relaxed and he's starting to know a little of my secrets.  I don't know too much about him but he's older so I'm sure he knows more of me than I know myself at this point.  So what am I benefiting from this, his company, someone to talk to, amend the loneliness, killing time, the attention?  I don't know but I am getting the short end of the stick because I can use this energy to meet someone new to where the possibilities can evolve into something more compared to his version of what we can evolve to.  So my question is can we be friends? Because it's not about male or female, married or single...but are you both benefiting from this friendship in a positive way and my answer is no.