No, you're not crazy for not having seen it. I love Walking Dead (the comics as well as the show), but it's far from being the best show on TV or anything like that. It started out exceptionally well with one of the best pilots I've ever seen, then sort of petered out over the rest of the first season (which was only six episodes), but had a good first season finale.
The second season started out strong as well, but the middle kind of, well, sucked. That's probably in large part because there were creative differences behind the scenes and Frank Darabont was fired from the show, but also because the writers didn't take any time in the first season trying to actually make us care about any characters, so by the time they were standing around talking about what they should do, you didn't really give a because there were basically no likable people in the group (with one or two noteworthy exceptions). Season two had a strong ending as well, though, and gave it some good momentum heading into season three.
Season three, so far, has been far and away the best and most consistent of the three seasons, largely because where the first and second seasons tended to veer from the plot of the comics a bit too much, this season is following it much more closely. I think they realized the comics are popular for a reason and decided to stick to the story a bit more. They've still changed a few things here and there, but it's a vast improvement in my opinion. It's still not a particularly well written show and there are some...let's say, mediocre...actors in the cast, but it's tense and exciting and is a show where literally no one is safe. Well, there are two characters who are 100% safe, but after that absolutely no one else is, and could die at any moment.
It's very, very gory, so I don't know if that'd make you more or less likely to watch. And a fan favorite character from the comics is being introduced in the next episode, played by a guy from The Wire, which makes me extremely giddy.