Prime Minister Tony Abbott has appeared to back a ban on the burqa being worn in Parliament House but at the same time said no-one has ever sought to enter the building "so attired".

But that claim appears to have been directly contradicted by one of Mr Abbott's senators who has spearheaded the campaign to have the controversial Islamic garment banned in Parliament.  Liberal senator Cory Bernardi says his intervention was sparked by seeing the face-covering veil being worn inside the building some years ago.

A final decision on a burqa ban in Parliament House could be made within a week, after Speaker Bronwyn Bishop asked ASIO and the federal police for an independent security assessment of the risks or otherwise of full facial coverings.

The Prime Minister said on Wednesday he wished that the burqa wasn't worn at all but said it was the choice of women to do so.

"Now I've said before I find it a fairly confronting form of attire and frankly I wish it weren't worn," he said.

"We are free country, we are a free society and it's not the business of government to tell people what they should and shouldn't wear,"

He cautioned against making a "mountain over a molehill" saying there is no record of the burqa ever being worn into the building.

"Has anyone ever sought entry to this building so attired? As far as I'm aware, no," he said.

"But I just want to stress that this is a secure building and it should be governed by the rules that are appropriate for a secure building and obviously people need to be identifiable in a secure building such as this," he said.

But Senator Bernardi told Fairfax Media his concerns were borne from having seen veiled people inside the building.

"What prompted my security concerns three years ago was when I did encounter a group of veiled individuals and inquired with security whether they were required to show their face before entering the building," he said.

"This then led to the extraordinary admission by parliamentary officials that anyone could enter with their face covered even if they were wearing a balaclava, this was a ridiculous position then and in light of the heightened security tensions it appears even more ridiculous now."

Asked about the apparent discrepancy between his own account and the Prime Minister's, Senator Bernardi said Parliament was a big building and not all parliamentarians would see the same visitors.

The newly-created Parliament House security taskforce, which comprises Ms Bishop, Senate president Stephen Parry and representatives of ASIO, the AFP and the departments of Prime Minister and Cabinet, Finance and Parliament Services met on Wednesday to discuss the issue.

Ms Bishop, who chaired the meeting, asked the two security agencies to provide her and Senator Parry with advice on the issue.

If the rules change only  veils that completely cover a person's face would be subject to a ban. The hijab, or head scarf, would not be subject to a ban if one were introduced.

A spokesman for the Speaker told Fairfax Media that it would be at least a week before Ms Bishop received advice back on the issue and made a decision.

At present, anyone entering a private area of Parliament House with a covered face has to uncover their face.

The rule change, if it proceeded, would require people with a covered face who entered a public area to uncover their face when entering the building.

The newly announced Commissioner of the AFP Andrew Colvin refused to say whether or not the burqa should be banned in Parliament House, which has been the subject of a major escalation of security in recent weeks.

"We need to be careful not to make something of an issue that police deal with each and every day in suburbs of Sydney and Melbourne," the incoming Commissioner said.

Mr Abbott's comments follows Fairfax Media's revelations that his most senior adviser, chief of staff Peta Credlin also supports a ban in the building on security grounds.

Labor MP Alannah MacTiernan said Ms Credlin's support for the ban had the potential to undermine Mr Abbott's call for calm over national security.

She said the campaign to ban the burqa from the building was a "pathetic attempt to whip up division" based on a "concocted problem" with the sole intent of "raising the temperature on an already difficult issue".

She said she had not once seen a burqa in Parliament House since her election in 2013, even when attending Islamic events.

"The last thing we need to do is go down this backwater, claiming we've got hundreds of people roaming the corridors of Parliament in burqas," Ms MacTiernan said.

Australian Muslim Women's Centre for Human Rights chairperson Tasneem Chopra said the burqa debate was a dangerous one because it highlighted differences within the Australian community at the expense of Australian Muslims.

"The community in general is feeling under siege. There is a constant demarcation between Muslim and Australian as if they can't coexist," she said.

Ms Chopra said the full-facial coverings weren't worn in Australia, and those just exposing a woman's eyes were rare.

"I am in Canberra now and in Parliament House, I have never seen a woman in a burqa, or a niqab," she said.

The debate about the burqa in Parliament House comes amid a controversial push from PUP senator Jacqui Lambie for a widespread ban on the burqa in public, which has been backed by Coalition backbenchers Cory Bernardi and George Christensen.

Opposition Leader Bill Shorten, who does not support a burqa ban in Parliament House, called on Mr Abbott on Wednesday to quieten the voices within his party "out there pushing socially divisive arguments".

"He should say that the Liberal government he leads will conduct itself in a bipartisan matter along with Labor to promote social cohesion."

Speaking at the National Press Club, Attorney-General George Brandis said: "I have no concerns with Muslims wearing the burqa and don't have a preference either way because frankly it's none of my business."

with Deborah Gough

Follow us on Twitter