Turning off Cloudflare should not cause any trouble. Cloudflare has a TTL of 5 minutes on its nameserver records, so within 5 minutes of gray-clouding a domain the domain records will be updated so browsers once again connect directly to your website.
There are some caveats however. If you use Flexible SSL or Full SSL (not Strict) and your server does not have a trusted HTTPS certificate properly deployed, then all HTTPS connections will break.
In addition, if you plan on turning off Cloudflare permanently and removing your domain from their services, a bit of planning is needed. First configure your nameservers outside of Cloudflare with all the A, AAAA, MX, etc. records that are currently in your Cloudflare account then switch over your domain records to use those nameservers. Then gray-cloud all your Cloudflare records to make them direct connect. Wait at least 48 hours before deleting your domain from Cloudflare, even if you see your new nameservers resolving your site. This is because it can take up to 48 hours for nameserver changes to propagate to ISPs worldwide.