"Kashiwa" of the term "kashiwabawami," written as 柏 in Japanese, represents plain wood (written as 白木 in Japanese) with one Chinese character (because the character 柏 can be divided into two parts: 木 and 白), and this has nothing to do with kashiwa (oak), a type of trees.
柏は白木を一つの漢字に直したもので植物のカシワとは関係ない。