The black bees are carpenter bees, and you should worry if you see a lot of holes in your timbers. Otherwise, you have nothing to worry about.
Black bees with a touch of yellow could be bumble bees. They are good for pollination.
My neighbor has a flowering tree that is right on the border of his backyard and ours. It attracts TONS of those big black bees! Should I be worried about them?